1. Field of the Invention
The present invention relates to a calibration method and device, a device for generating calibration data and a method thereof, and information providing medium; whereby information regarding the position of an object in 3-dimensional space is calculated from 2-dimensional images obtained by imaging devices which perform imaging of the object.
2. Description of the Related Art
There is a known stereo camera system serving as a 3-dimensional position detecting device, in which a plurality (e.g., two) of video cameras (hereafter simply referred to as xe2x80x9ccamerasxe2x80x9d) are used to comprise a so-called stereo camera by which an object in a 3-dimensional space is imaged and the position of the object in the 3-dimensional space is determined based on the 2-dimensional images obtained by the imaging.
Details of a stereo camera system are disclosed in detail in MASATOSHI OKUTOMI and TAKEO KANEIDE: xe2x80x9cStereo Matching Using Multiple Base-Line Lengthsxe2x80x9d, Institute of Electronics, Information and Communication Engineers Journal D-II, Vol. J75-D-II. No. 8, pp. 1317-1327 (August 1992), and so forth.
With a stereo camera system, an object whose positional information in 3-dimensional space is to be obtained is imaged by multiple cameras, and information regarding the position of the object in 3-dimensional space can be obtained from the positional information of the object projected on a light-receiving plane (hereafter referred to as xe2x80x9cscreenxe2x80x9d) of the photo-electric converting devices (e.g., CCD) of each camera. Accordingly, in the event that there is positional information of an object existing at a certain position within a 3-dimensional space, and an object at that position, the correlated relation with the positional information of the object projected on the screen of each camera (correlated relation of position information) must be determined beforehand. The process of obtaining this correlated relation of position information is referred to as xe2x80x9ccalibrationxe2x80x9d, and is carried out by a calibration device.
FIG. 8 is an external perspective view of a known calibration device for performing calibration. In FIG. 8, pipes 106 and 107 are included in the same plane in a 3-dimensional space, and a carriage 108 is provided so as to smoothly move along the pipes 106 and 107. Attached to the carriage 108 is a stereo camera, comprised of a first camera 101 and a second camera 102 which have been integrally joined by means of a metal piece 103.
The pipes 106 and 107 are inscribed with scale marks, so as to enable measuring the distance that the carriage 108 slides. A plate 109 with a square lattice-work pattern drawn thereupon is provided in a direction perpendicular to the direction of sliding of the carriage 108. The horizontal direction of the square lattice-work serves as the X-axis, the vertical direction thereof as the Y-axis, and the direction of sliding, i.e., the direction perpendicular to the square lattice-work is the Z-axis. Z greater than 0 holds for the side of the plate 109 on which the camera is provided. Such a 3-dimensional coordinates system with the X-axis, Y-axis, and Z-axis, is defined as a xe2x80x9cworld coordinatesxe2x80x9d system.
Calibration measurement is performed by shifting the carriage 108 carrying the aforementioned stereo camera along the Z-axis, and imaging the plate 109 from two positions. FIG. 9 is a diagram describing a case in which imaging is performed from two positions, viewing the device shown in FIG. 8 from directly above.
First, the first camera 101 and the second camera 102 are fixed at a certain position P1, the plate 109 is imaged such that the square lattice-work pattern is recorded, following which the first camera 101 and the second camera 102 are slid along the Z-axis to another position P2 by a distance of M by means of sliding the carriage 108, and the plate 109 is imaged once more. Here, FIG. 9 shows the first camera 101 and the second camera 102 being moved in a direction away from the plate 109, but this direction of sliding may be reversed.
Thus, the 2-dimensional images obtained by means of sliding a stereo camera comprised of the first camera 101 and the second camera 102 and imaging the plate 109 can also be obtained by means of an arrangement wherein the stereo camera is fixed and the plate 109 is shifted instead as well, as shown in FIG. 10.
That is to say, the same 2-dimensional images can be obtained by an arrangement such as shown in FIG. 10, wherein the first camera 101 and the second camera 102 are fixed to a certain position P1, the plate 109 is imaged such that the square lattice-work pattern is recorded, following which the plate 109 is slid along the Z-axis by a distance of M to another position P2 in a direction away from the first camera 101 and the second camera 102, and the plate 109 is imaged once more at that position.
In FIG. 10, with the lower left corner of the square lattice-work pattern drawn on the plate before moving the plate by a distance of M (first square lattice-work pattern Q1) serving as the origin point and also as the origin for the world coordinates system, the position (i, j) on the first square lattice-work pattern Q1 for the plate 109 is (i, j, O) on the world coordinates. Also, the position (i, j) on the second square lattice-work pattern Q2 after the plate 109 has been shifted by a distance of M is (i, j, xe2x88x92M) on the world coordinates.
FIG. 11 shows the first camera 101, and the first square lattice-work Q1 and second square lattice-work Q2 on the plate 109. The optical center of the first camera 101 is 01, and positional information of the object is case upon the screen serving as the light-receiving surface of a CCD 122 or the like. For example, let us say that coordinates position (p, q) at the first square lattice-work Q1 is projected, and coordinates position (r, s) at the second square lattice-work Q2 is projected. Incidentally, the coordinates of positions other than upon the vertical and horizontal lines in the grid can be calculated by interpolation.
Giving the same description once more using world coordinates, 3-dimensional coordinates positions (p, q, O) and (r, s, xe2x88x92M) are projected onto the coordinates position (h, k) on the CCD 122. That is, in the event that the 2-dimensional coordinates position (h, k) and the 3-dimensional coordinates positions (p, q, O) and (r, s, xe2x88x92M) are connected by a line N, all points on this line N are projected onto the coordinates position (h, k) on the CCD 122.
Accordingly, the line N represents a correlated relation (correlated relation of position information) between positional information of objects in a 3-dimensional space (in this case, coordinates in the world coordinates system), and 2-dimensional information obtained by imaging the object (in this case, coordinates on the 2-dimensional coordinates system on the CCD 122).
This line N can be calculated as follows:
(xxe2x88x92r)/(pxe2x88x92r)=(yxe2x88x92s)/(qxe2x88x92s)=(z+M)/M
In the same manner as calculating line N, lines projected on other coordinates positions as collections of points in the 3-dimensional space are also calculated for the other 2-dimensional coordinate systems on the CCD, as well. The same is also carried out with the second camera 102.
Thus, by calculating all lines for the first camera 101 and the second camera 102, calibration of the stereo camera system is completed.
The positional information of an object in the 3-dimensional space can be calculated as follows, using a stereo camera system which has been calibrated as described above.
First, an object is imaged using the stereo camera. For example, let us say that the object 127 shown in FIG. 127 is projected at the position (a, b) on the screen 122 of the first camera 101 and the position (c, d) on the screen 128 of the second camera 102. The lines 130 and 131 on the world coordinates system corresponding with the positions (a, b) and (C, d) have already been determined in the calibration (initialization) of the above calibration device, so it is possible to calculation the point of intersection of these lines on the world coordinates system. Thus, the positional of an object in 3-dimensional space can be measured.
Summarizing the above: first, measurement is made regarding at which position on the screens 122 and 128 of the first camera 101 and the second camera 102 the object has been projected. Next, the point of intersection of the lines 130 and 131 which each correspond with each of the positions of projection is calculated on the world coordinates system. This point of intersection is the position of the object on the world coordinates system, i.e., in the 3-dimensional space.
Now, with the calibration device shown in FIG. 8, with the distance L between the first and second cameras 101 and 102 and the plate 109 as several meters, an area around 4 meters by 3 meters becomes necessary for the area on which the square lattice-work pattern is made on the plate 109. The reason is: in the event that the area on which the square lattice-work pattern is made is small, the square lattice-work pattern is not projected on the periphery of the CCD (screen) 122 shown in FIG. 11 when being imaged by the camera, so the line N regarding pixels at the periphery thereof cannot be calculated.
However, it has been difficult to fabricate a precise square lattice-work pattern of such a size as 4 meters by 3 meters. Accordingly, only square lattice-work pattern with a certain degree of warping could be provided, and consequently, highly precise calibration could not be carried out.
The present invention has been made in light of the above-described present state, and accordingly, it is an object of the present invention to provide a calibration method and device, capable of performing highly precise calibration using a reference object provided with square lattice-work pattern of a size which can be fabricated with precision.
It is another object of the present to provide a method for generating data for calibration, capable of performing highly precise calibration using a reference object provided with square lattice-work pattern of a size which can be fabricated with precision.
According to one aspect of the present invention, a calibration method for calibrating an imaging device for determining positional information of an object in a 3-dimensional space based on 2-dimensional images obtained from the imaging device conducts calibration of the imaging device based on 3-dimensional coordinates at least two positions of a reference object set within a 3-dimensional space confined to the range in which the imaging device performs imaging, and image data obtained by the imaging device performing imaging of the reference object at each set position.
According to another aspect of the present invention, a calibration device for calibrating an imaging device for determining positional information of an object in a 3-dimensional space based on 2-dimensional images obtained from the imaging device, comprises a reference object set in a 3-dimensional space within the range in which the imaging device performs imaging of the object, 3-dimensional position information measuring means for measuring the 3-dimensional coordinates position of the reference object, and control means for generating calibration information for the imaging device, based on 3-dimensional coordinates at least two positions of a reference object set within a 3-dimensional space confined to the range in which the imaging device performs imaging, and image data obtained by the imaging device performing imaging of the reference object at each set position.
According to a further aspect of the present invention, a calibration data generating method for generating data for calibrating an imaging device which determines positional information of an object in a 3-dimensional space based on 2-dimensional images obtained from the imaging device comprises the steps of: receiving data indicating 3-dimensional position information of the reference object which has been measured multiple times by a 3-dimensional position information measuring device; receiving image data of the pattern which has been measured multiple times by the imaging device; and generating data for calibration for obtaining a 3-dimensional space position projected on each position of the screens of the imaging device, based on both types of data.
According to a yet another aspect of the present invention, a calibration data generating device for generating data for calibrating an imaging device which determines positional information of an object in a 3-dimensional space based on 2-dimensional images obtained from the imaging device comprises: a first receiving unit for receiving data indicating 3-dimensional position information of the reference object which has been measured multiple times by a 3-dimensional position information measuring device; a second receiving unit for receiving image data of the pattern which has been measured multiple times by the imaging device; and a control unit for generating data for calibration for obtaining a 3-dimensional space position projected on each position of the screens of the imaging device, based on both types of data.
According to a still another aspect of the present invention, an information providing medium which provides control information for calibrating an imaging device for determining positional information of an object in a 3-dimensional space based on 2-dimensional images obtained from the imaging device conducts calibration of the imaging device based on 3-dimensional coordinates at least two positions of a reference object set within a 3-dimensional space confined to the range in which the imaging device performs imaging, and image data obtained by the imaging device performing imaging of the reference object at each set position.